DeProCams: Simultaneous Relighting, Compensation and Shape Reconstruction for Projector-Camera Systems

Image-based relighting, projector compensation and depth/normal reconstruction are three important tasks of projector-camera systems (ProCams) spatial augmented reality (SAR). Although they share a similar pipeline finding image mappings, in tradition, addressed independently, sometimes with different prerequisites, devices sampling images. In practice, this may be cumbersome for SAR applications to address them one-by-one. paper, we propose novel end-to-end trainable model named DeProCams explicitly learn the photometric geometric mappings ProCams, once trained, can applied simultaneously tasks. decomposes into subprocesses: shading attributes estimation, rough direct light estimation photorealistic neural rendering. A particular challenge by is occlusion, which exploit epipolar constraint differentiable mask. Thus, it learned along other modules. Afterwards, improve convergence, apply constraints such that intermediate results plausible. our experiments, shows clear advantages over previous arts promising quality meanwhile being fully differentiable. Moreover, solving unified model, waives need additional optical devices, radiometric calibrations structured light.